Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds

We discuss methods for the evaluation of probabilistic predictions of vector-valued quantities, that can take the form of a discrete forecast ensemble or a density forecast. In particular, we propose a multivariate version of the univariate verification rank histogram or Talagrand diagram that can be used to check the calibration of ensemble forecasts. In the case of density forecasts, Box’s density ordinate transform provides an attractive alternative. The multivariate energy score generalizes the continuous ranked probability score. It addresses both calibration and sharpness, and can be used to compare deterministic forecasts, ensemble forecasts and density forecasts, using a single loss function that is proper. An application to the University This invited paper is discussed in the comments available at: http://dx.doi.org/10.1007/s11749-008-0115-9, http://dx.doi.org/10.1007/s11749-008-0116-8, http://dx.doi.org/10.1007/s11749-008-0117-7, http://dx.doi.org/10.1007/s11749-008-0118-6, http://dx.doi.org/10.1007/s11749-008-0119-5, http://dx.doi.org/10.1007/s11749-008-0120-z, http://dx.doi.org/10.1007/s11749-008-0121-y. T. Gneiting ( ) · L.I. Stanberry Department of Statistics, University of Washington, Seattle, WA, USA e-mail: [email protected] L.I. Stanberry e-mail: [email protected] E.P. Grimit 3Tier Environmental Forecast Group, Seattle, WA, USA e-mail: [email protected] L. Held Institut für Sozialund Präventivmedizin, Abteilung Biostatistik, Universität Zürich, Zürich, Switzerland e-mail: [email protected] N.A. Johnson Department of Statistics, Stanford University, Stanford, CA, USA e-mail: [email protected] 212 T. Gneiting et al. of Washington mesoscale ensemble points at strengths and deficiencies of probabilistic short-range forecasts of surface wind vectors over the North American Pacific Northwest.